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(54) Method and apparatus for automated image analysis of biological specimens 



(57) A method and apparatus for automated cell 
analysis of biological specimens automatically scans at 
a low magnification to acquire images (288) which are 
analyzed to determine candidate cell objects of interest. 
The low magnification images are converted from a first 
color space to a second color space (290). The color 
space converted image is then low pass filtered (292) 
and compared to a threshold (294) to remove artifacts 
and background objects from the candidate object of in- 
terest pixels of the color converted image. The candi- 
date object of interest pixels are morphologically proc- 
essed (296) to group candidate object of interest pixels 
together into groups which are compared to blob param- 



eters (298)to identify candidate objects of interest which 
correspond to cells or other structures relevant to med- 
ical diagnosis of the biological specimen. The location 
coordinates of the objects of interest are stored and ad- 
ditional images of the candidate cell objects are ac- 
quired at high magnification. The high magnification im- 
ages are analyzed in the same manner as the low mag- 
nification images to confirm the candidate objects of in- 
terest which are objects of interest A high magnification 
image of each confirmed object of interest is stored for 
later review and evaluation by a pathologist. 
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Description 

Background of the Invention 

[0001] In the field of medical diagnostics including on- 
cology, the detection, identification, quantitation and 
characterization of cells of interest, such as cancer cells, 
through testing of biological specimens is an important 
aspect of diagnosis. Typically, a biological specimen 
such as bone marrow, lymph nodes, peripheral blood, 
cerebrospinal fluid, urine, effusions, fine needle aspi- 
rates, peripheral blood scrapings or other materials are 
prepared by staining the specimen to identify cells of in- 
terest. One method of ceii specimen preparation is to 
react a specimen with a specific probe which can be a 
monoclonal antibody, a polyclonal antiserum, or a nu- 
cleic acid which is reactive with a component of the cells 
of interest, such as tumor cells. The reaction may be 
detected using an enzymatic reaction, such as alkaline 
phosphatase or glucose oxidase or peroxidase to con- 
vert a soluble colorless substrate to a colored insoluble 
precipitate, or by directly conjugating a dye to the probe. 
[0002] Examination of biological specimens in the 
past has been performed manually by either a lab tech- 
nician or a pathologist. In the manual method, a slide 
prepared with a biological specimen is viewed at a low 
magnification under a microscope to visually locate can- 
didate cells of interest. Those areas of the slide where 
cells of interest are located are then viewed at a higher 
magnification to confirm those objects as cells of inter- 
est, such as tumor or cancer cells. The manual method 
is time consuming and prone to error including missing 
areas of the slide. 

[0003] Automated cell analysis systems have been 
developed to improve the speed and accuracy of the 
testing process. One known interactive system includes 
a single high power microscope objective for scanning 
a rack of slides, portions of which have been previously 
identified for assay by an operator. In that system, the 
operator first scans each slide at a low magnification 
similar to the manual method and notes the points of 
interest on the slide for later analysis. The operator then 
stores the address of the noted location and the asso- 
ciated function in a data fife. Once the points of interest 
have been located and stored by the operator, the slide 
is then positioned in an automated analysis apparatus 
which acquires images of the slide at the marked points 
and performs an image analysis. 
[0004] EP^-0 213 666 discloses a method and ap- 
paratus for displaying a single parameter image, in par- 
ticular a grayscale image. Each pixel of the original im- 
age can have a contrast range represented by 8-32 Bits. 
In order to make this large contrast range visible to the 
user, it is taught to map a compressed local signal com- 
ponent of the original image into an intensity informa- 
tion, and a background signal component into a color 
information of a displayed image. 



Summary of the Invention 

[0005] A problem with the foregoing automated sys- 
tem is the continued need for operator input to initially 
5 locate cell objects for analysis. Such continued depend- 
ence on manual input can lead to errors including cells 
of interest being missed. Such errors can be critical es- 
pecially in assays for so-called rare events, e.g., finding 
one tumor cell in a cell population of one million normal 
10 cells. Additionally, manual methods can be extremely 
time consuming and can require a high degree of train- 
ing to properly identify and/or quantify cells. This is not 
only true for tumor cell detection, but also for other ap- 
plications ranging from neutrophil alkaline phosphatase 
is assays, reticulocyte counting and maturation assess- 
ment, and others. The associated manual labor leads to 
a high cost for these procedures in addition to the po- 
tential errors that can arise from long, tedious manual 
examinations. A need exists, therefore, for an improved 
20 automated cell analysis system which can quickly and 
accurately scan large amounts of biological material on 
a slide. Accordingly, the present invention provides a 
method for automated cell analysis which eliminates the 
need for operator input to locate cell objects for analysis. 
25 [0006] In accordance with the present invention, a 
method of automatic analysis of a color image in a first 
color space of a slide having a biological specimen pre- 
pared with a reagent comprises the steps listed in claim 
1. For carrying out the inventive method, a slide pre- 
30 pared with a biological specimen and reagent is placed 
in a slide carrier which preferably holds four slides. The 
slide carriers are loaded into an input hopper of the au- 
tomated system. The operator may then enter data iden- 
tifying the size, shape and location of a scan area on 
35 each slide, or, preferably, the system automatically lo- 
cates a scan area for each slide during slide processing. 
The operator then activates the system for slide 
processing. At system activation, a slide carrier is posi- 
tioned on an X-Y stage of an optical system. Any bar 
43 codes used to identify slides are then read and stored 
for each slide in a carrier. The entire slide is rapidly 
scanned at a low magnification, typically 10x. At each 
location of the scan, a low magnification image is ac- 
quired and processed to detect candidate objects of in- 
45 terest. Preferably, color, size and shape are used to 
identify objects of interest. The location of each candi- 
date object of interest is stored. 
[0007] At the completion of the low level scan for each 
slide in the carrier on the stage, the optical system is 
so adjusted to a high magnification such as40xor60x, and 
the X-Y stage is positioned to the stored locations for 
the candidate objects of interest on each slide in the car- 
rier. A high magnification image is acquired for each 
candidate object of interest and a series of image 
55 processing steps are performed to confirm the analysis 
which was performed at low magnification. A high mag- 
nification image is stored for each confirmed object of 
interest. These images are then available for retrieval 
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by a pathologist or cytotechnologist to review for final 
diagnostic evaluation. Having stored the location of 
each object of interest, a mosaic comprised of the can- 
didate objects of interest for a slide may be generated 
and stored. The pathologist or cytotechnologist may 
view the mosaic or may also directiy view the slide at 
the location of an object of interest in the mosaic for fur- 
ther evaluation. The mosaic may be stored on magnetic 
media for future reference or may be transmitted to a 
remote site for review and/or storage. The entire proc- 
ess involved in examining a single slide takes on the 
order of 2-15 minutes depending on scan area size and 
the number of detected candidate objects of interest. 
[0008] The present invention has utility in the field of 
oncology forthe eariy detection of minimal residua! dis- 
ease ("micrometastases"). Other useful applications in- 
clude prenatal diagnosis of fetal ceils in maternal blood 
and in the field of infectious diseases to identify patho- 
gens and viral loads, alkaline phosphatase assess- 
ments, reticulocyte counting, and others. 
[0009] The processing of images acquired in the au- 
tomated scanning according to the present invention in- 
cludes the step of transforming the image to a different 
color space. The processing of images furthermore pref- 
erably includes the steps of filtering the transformed im- 
age with a low pass filter; dynamically thresholding the 
pixels of the filtered image to suppress background ma- 
terial; performing a morphological function to remove ar- 
tifacts fromthethresholded image; analyzing thethresh- 
olded image to determine the presence of one or more 
regions of connected pixels having the same color; and 
categorizing every region having a size greater than a 
minimum size as a candidate object of interest. 
[0010] Forcarryingoutthe inventive method, the scan 
area may be automatically determined by scanning the 
slide; acquiring an image at each slide position; analyz- 
ing texture information of each image to detect the edg- 
es of the specimen; and storing the locations corre- 
sponding to the detected edges to define the scan area. 
[0011] An automated focusing of the optical system 
for carrying out the inventive method is achieved by in- 
itially determining a focal plane from an array of points 
or locations in the scan area. The derived focal plane 
enables subsequent rapid automatic focusing in the low 
power scanning operation. The focal plane is deter- 
mined by determining proper focal positions across an 
array of locations and performing an analysis such as a 
least squares fit of the array of focal positions to yield a 
focal plane across the array. Preferably, a focal position 
at each location is determined by incrementing the po- 
sition of a Z stage for a fixed number of coarse and fine 
iterations. At each iteration, an image is acquired and a 
pixel variance or other optical parameter about a pixel 
mean forthe acquired image is calculated to form a set 
of variance data. A least squares fit is performed on the 
variance data according to a known function. The peak 
value of the least squares fit curve is selected as an es- 
timate of the best focal position. 



[0012] Another focal position method for high magni- 
fication locates a region of interest centered about a 
candidate object of interest within a slide which were lo- 
cated during an analysis of the low magnification imag- 
5 es. The region of interest is preferably n columns wide, 
where n is a power of 2. The pixels of this region are 
then processed using a Fast Fourier Transform to gen- 
erate a spectra of component frequencies and corre- 
sponding complex magnitude for each frequency com- 
10 ponent. Preferably, the complex magnitude of the fre- 
quency components which range from 25% to 75% of 
the maximum frequency component are squared and 
summed to obtain the total power for the region of inter- 
est. This process is repeated for other Z positions and 
15 the Z position corresponding to the maximum total pow- 
er forthe region of interest is selected as the best focal 
position. This process is preferably used to select a Z 
position for regions of interest for slides containing neu- 
trophils stained with Fast Red to identify alkaline phos- 
20 phatase in cell cytoplasm and counferstained with he- 
motoxylin to identify the nucleus of the neutrophil cell. 
This focal method may be used with other stains and 
types of biological specimens, as welf. 
[0013] For carrying out the inventive method a slide 
is mounted onto a slide carrier with a number of other 
slides side-by-side. The slide carrier is positioned in an 
input feeder with other slide carriers to facilitate auto- 
matic analysis of a batch of slides. The slide carrier is 
loaded onto the X-Y stage of the optical system for the 
analysis of the slides thereon. Subsequently, the first 
slide carrier is unloaded into an output feeder after au- 
tomatic image analysis and the next carrier is automat- 
ically loaded. 

Brief Description of the Drawings 

[0014] The above and other features of the invention 
including various novei details of construction and com- 
binations of parts will now be more particularly de- 
scribed with reference to the accompanying drawings 
and pointed out in the claims. It will be understood that 
the particular apparatus for carrying out the invention is 
shown by way of illustration only and not as a limitation 
of the invention. The principles and features of this in- 
vention may be employed in varied and numerous em- 
bodiments without departing from the scope of the in- 



vention. 




Fig. 1 


is a perspective view of an apparatus for 




automated ceil analysis. 


Fig. 2 


is a block diagram of the apparatus 




shown in Fig. 1. 


Fig. 3 


is a block diagram of the microscope 




controller of Fig. 2. 


Fig.4 


is a plan view of the apparatus of Fig. 1 




having the housing removed. 


Fig. 5 


is a side view of a microscope subsystem 




of the apparatus of Fig. 1. 
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Fig. 6a is a top view of a slide carrier for use with 

the apparatus of Fig. 1. 
Fig. 6b isabottomviewoftheslidecarrierofFig. 

6a. 

Fig. 7a is a top view of an automated slide han- 

dling subsystem of the apparatus of Fig. 
1. 

Fig. 7b is a partial cross-sectional view ofthe au- 

tomated slide handling subsystem of Fig. 
7a taken on line A-A. 

Fig. 8 is an end view ofthe input module ofthe 

automated slide handling subsystem. 

Figs. 8a-8d illustrate the input operation ofthe auto- 
matic slide handling subsystem. 

Figs. 9a-9d illustrate the output operation of the au- 
tomated slide handling subsystem. 

Fig. 10 is a flow diagram ofthe procedure for au- 

tomatically determining a scan area. 

Fig. 1 1 shows the scan path on a prepared slide 

in the procedure of Fig. 10. 

Fig. 12 illustrates an image of afield acquired in 

the procedure of Fig. 10. 

Fig. 13A is a flow diagram of a preferred proce- 
dure for determining a focal position. 

Fig. 13B is a flow diagram of a preferred proce- 
dure for determining a focal position for 
neutrophils stained with Fast Red and 
counterstained with hemotoxylin. 

Fig. 14 is a flow diagram of a procedure for au- 

tomatically determining initial focus. 

Fig. 15 shows an array of slide positions for use 

in the procedure of Fig. 14. 

Fig. 16 is a flow diagram of a procedure for au- 

tomatic focusing at a high magnification. 

Fig. 17A is a flow diagram of an overview of the 
preferred process to locate and identify 
objects of interest in a stained biological 
specimen on a slide. 

Fig. 17B is a flow diagram of a procedure forcolor 
space conversion. 

Fig. 18 is a flow diagram of a procedure forback- 

ground suppression via dynamic thresh- 
olding. 

Fig. 19 is a flow diagram of a procedure for mor- 

phological processing. 

Fig. 20 is a flow diagram of a procedure for blob 

analysis. 

Fig. 21 is a flow diagram of a procedure for im- 

age processing at a high magnification. 

Fig. 22 illustrates a mosaic of cell images pro- 

duced by the apparatus. 

Fig. 23 is a flow diagram of a procedure for es- 

timating the number of nucleated cells in 
a scan area. 

Fig. 24 illustrates the apparatus functions avail- 

able in a user interface ofthe apparatus. 



Detailed Description ofthe Preferred Embodiment 

[0015] Referring now to the figures, an apparatus for 
automated cell analysis of biological specimens is gen- 

s eraily indicated by reference numeral 10 as shown in 
perspective view in Fig. 1 and in biock diagram form in 
Fig. 2. The apparatus 10 comprises a microscope sub- 
system 32 housed in a housing 12. The housing 12 in- 
cludes a slide carrier input hopper 16 and a slide carrier 

10 output hopper 18. A door 14 in the housing 12 secures 
the microscope subsystem from the external environ- 
ment. A computer subsystem comprises a computer 22 
having a system processor 23, an image processor 25 
and a communications modem 29. The computer sub- 

15 system further includes a computer monitor 26 and an 
image monitor27 and other external peripherals includ- 
ing storage device 21 , track ball device 30, keyboard 28 
and color printer35. An external power supply 24 is also 
shown for powering the system. Viewing oculars 20 of 

20 the microscope subsystem project from the housing 12 
for operator viewing. The apparatus 10 further includes 
a CCD camera 42 for acquiring images through the mi- 
croscope subsystem 32. A microscope controller 31 un- 
der the control of system processor 23 controls a 

25 number of microscope-subsystem functions described 
further in detail. An automatic slide feed mechanism 37 
in conjunction withX-Ystage 38 provide automatic slide 
handling in the apparatus 10. An illumination light 
source 48 projects light onto theX-Y stage 38 which is 

30 subsequently imaged through the microscope subsys- 
tem 32 and acquired through CCD camera 42 for 
processing in the image processor 25. A Z stage or fo- 
cus stage 46 under control ofthe microscope controller 

31 provides displacement ofthe microscope subsystem 
35 in the Z plane for focusing. The microscope subsystem 

32 further includes a motorized objective turret 44 for 
selection of objectives. 

[0016] The purpose ofthe apparatus 10 is forthe un- 
attended automatic scanning of prepared microscope 

to slides for the detection and counting of candidate ob- 
jects of interest such as normal and abnormal cells, e. 
g. s tumor cells. The apparatus 1 0 may be utilized for rare 
event detection in which there may be only one candi- 
date object of interest per several hundred thousand 

45 normal cells, e.g., one to five candidate objects of inter- 
est per 2 square centimeter area ofthe slide. The appa- 
ratus 1 0 automatically locates and counts candidate ob- 
jects of interest and estimates normal cells present in a 
biological specimen on the basis of color, size and 

so shape characteristics. A number of stains are used to 
preferential ly stain cand idate o bjects of interest a nd nor- 
mal cells different colors so that such cells can be dis- 
tinguished from each other. 

[0017] As noted in the background of the invention, a 
55 biological specimen may be prepared with a reagent to 
obtain a colored insoluble precipitate. The apparatus is 
used to detect this precipitate as a candidate object of 
interest. 
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[0018] During operation of the apparatus 1 0, a pathol- 
ogist or laboratory technician mounts prepared slides 
onto slide carriers. A slide carrier 60 is illustrated in Fig. 
8 and will be described further below. Each slide carrier 
holds up to 4 slides. Up to 25 slide carriers are then load- 
ed - inio input hopper 16. The operator can specify the 
size, shape and location of the area to be scanned or 
alternatively, the system can automatically locate this 
area. The operator then commands the system to begin 
automated scanning of the slides through a graphical 
user interface. Unattended scanning begins with the au- 
tomatic loading of the first carrierand slide onto the pre- 
cision motorized X-Y stage 38. A barcode label affixed 
to the slide is read by a bar code reader 33 during this 
loading operation. Each slide is then scanned at a user 
selected low microscope magnification, for example, 
1 0x, to identify cand idate cells based on their color, size 
and shape characteristics. The X-Y locations of candi- 
date cells are stored until scanning is completed. 
[0019] After the low magnification scanning is com- 
pleted, the apparatus automatically returns to each can- 
didate cell, reimages and refocuses at a higher magni- 
fication such as 40x and performs further analysis to 
confirm the cell candidate. The apparatus stores an im- 
age of the cell for later reviewby a pathologist. All results 
and images can be stored to a storage device 21 such 
as a removable hard drive or DAT tape or transmitted to 
a remote site for review or storage. The stored images 
for each slide can be viewed in a mosaic of images for 
further review, in addition, the pathologist or operator 
can also directly view a detected cell through the micro- 
scope using the included oculars 20 or on image monitor 
27. 

[0020] Having described the overall operation of the 
apparatus 10 from a high level, the further details of the 
apparatus will now be described. Referring to Fig. 3, the 
microscope controller 31 is shown in more detail. The 
microscope controller 31 includes a number of subsys- 
tems connected through a system bus. A system proc- 
essor 102 controls these subsystems and is controlled 
by the apparatus system processor 23 through an RS 
232 controller 110. The system processor 102 controls 
a set of motor - control subsystems 114 through 124 
which control the input and output feeder, the motorized 
turret 44, the X-Y stage 38, and the Z stage 46 (Fig. 2). 
A histogram processor 108 receives input from CCD 
camera 42 for computing variance data during the fo- 
cusing operation described further herein. 
[0021] The system processor 102 further controls an 
illumination controller 106 for control of substage illumi- 
nation 48. The light output from the halogen light bulb 
which supplies illumination for the system can vary over 
time due to bulb aging, changes in optical alignment, 
and other factors. In addition, slides which have been 
"over stained" can reduce the camera exposure to an 
unacceptable level. In orderto compensate forthese ef- 
fects, the illumination controller 106 is included. This 
controller is used in conjunction with light control soft- 



ware to compensate for the variations in light level. The 
light control software samples the output from the cam- 
era at intervals (such as between loading of slide carri- 
ers), and commands the controller to adjust the light lev- 
s el to the desired levels. In this way, light control is auto- 
matic and transparent to the user and adds no additional 
time to system operation. 

[0022] The system processor 23 is preferably com- 
prised of dual parallel Intel Pentium 90 MHz devices. 
to The image processor 25 is preferably a Matrox Imaging 
Series 640 model. The microscope controller system 
processor 102 is an Advanced Micro Devices AMD29K 
device. 

[0023] Referring now to Figs. 4 and 5, further detail of 
15 the apparatus 10 is shown. Fig. 4 shows a plan view of 
theapparatus 10withthehousing 12 removed. A portion 
of the automatic slide feed mechanism 37 is shown to 
the left of the microscope subsystem 32 and includes 
slide carrier unloading assembly 34 and unloading plat- 
20 form 36 which in conjunction with slide carrier output 
hopper 18 function to receive slide carriers which have 
been analyzed. 

[0024] Vibration isolation mounts 40, shown in further 
detail in Fig. 5, are provided to isolate the microscope 

25 subsystem 32 from mechanical shock and vibration that 
can occur in a typical laboratory environment. In addition 
to external sources of vibration, the high speed opera- 
tion of the X-Y stage 38 can induce vibration into the 
microscope subsystem 32. Such sources of vibration 

30 can be isolated from the electro-optical subsystems to 
avoid any undesirable effects on image quality. The iso- 
lation mounts 40 comprise a spring 40a and piston 40b 
submerged in a high viscosity silicon gel which is en- 
closed in an elastomer membrane bonded to a casing 

35 to achieve damping factors on the order of 1 7 to 20%. 
[0025] The automatic slide handling feature will now 
be described. The automated slide handling subsystem 
operates on a single slide carrierat a time. A slide carrier 
60 is shown in Figs. 6a and 6b which provide a top view 

40 and a bottom view respectively. The slide carrier 60 in- 
cludes up to four slides 70 mounted with adhesive tape 
62. The carrier 60 includes ears 64 for hanging the car- 
rier in the output hopper 18. An undercut 66 and pitch 
rack 68 are formed at the top edge of the slide carrier 

45 60 for mechanical handling of the slide carrier. A keyway 
cutout 65 is formed in one side of the carrier 60 to facil- 
itate carrier alignment. A prepared slide 72 mounted on 
the slide carrier 60 includes a sample area 72a and a 
bar code label area 72b. 

so [0026] Fig. 7a provides a top viewofthe slide handling 
subsystem which comprises a slide input module 15, a 
slide output module 17 and X-Y stage drive belt 50. Fig. 
7b provides a partial cross-sectional view taken along 
line A-A of Fig. 7a. 

55 The slide input module 1 5 comprises a slide carrier in put 
hopper 16, loading platform 52 and slide carrier loading 
subassembly 54. The input hopper 16 receives a series 
of slide carriers 60 (Figs. 6a and 6b) in a stackon loading 
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platform 52. A guide key 57 protrudes from a side of the 
input hopper 16 to which the keyway cutout 65 (Fig. 6a) 
of the carrier is fit to achieve proper alignment. 
[0027] The input moduie 1 5 further includes a revolv- 
ing indexing cam 56 and a switch 90 mounted in the 
loading platform 52, the operation of which is described 
further below. The carrier loading subassembly 54 com- 
prises an infeed drive belt 59 driven by a motor 86. The 
infeed drive belt 59 includes a pusher tab 58 for pushing 
the slide carrier horizontally toward the X-Y stage 38 
when the belt is driven. A homing switch 95 senses the 
pusher tab 58 during a revolution of the belt 59. 
[0028] Referring specifically to Fig. 7a, the X-Y stage 
38 is shown with x position and y position motors 96 and 
97 respectively which are controlled by the microscope 
controller 31 (Fig. 3) and are not considered part of the 
slide handling subsystem. The X-Y stage 38 further in- 
cludes an aperture 55 for allowing illumination to reach 
the slide carrier. A switch 91 is mounted adjacent the 
aperture 55 for sensing contact with the carrier and 
thereupon activating a motor 87 to drive stage drive belt 
50 (Fig. 7b). The drive belt 50 is a double sided timing 
belt having teeth for engaging pitch rack 68 ofthe carrier 
60 {Fig. 6b). 

[0029] The slide output module 17 includes slide car- 
rier output hopper 18, unloading platform 36 and slide 
carrier unloading subassembly 34. The unloading sub- 
assembly 34 comprises a motor 89 for rotating the un- 
loading platform 36 about shaft 98 during an unloading 
operation described further below. An outfeed gear 93 
driven by motor 88 rotatably engages the pitch rack 68 
ofthe carrier 60 (Fig. 6b) to transport the carrierto a rest 
position against switch 92. A springloaded hold-down 
mechanism holds the carrier in place on the unloading 
platform 36. 

[0030] The slide handling operation will now be de- 
scribed. Referring to Fig. 8, a series of slide carriers 60 
are shown stacked in input hopper 16 with the top edges 
60a aligned. As the slide handling operation begins, the 
indexing cam 56 driven by motor 85 advances one rev- 
olution to allow only one slide carrierto drop to the bot- 
tom ofthe hopper 16 and onto the loading platform 52. 
[0031] Figs. 8a-8d showthe cam action in more detail. 
The cam 56 includes a hub 56a to which are mounted 
upper and lower leaves 56b and 5Qc respectively. The 
leaves 56b, 56c are semicircular projections oppositely 
positioned and spaced apart vertically. In a first position 
shown in Fig. 8a, the upper leaf 56b supports the bottom 
carrier at the undercut portion 66. At a position ofthe 
cam 56 rotated 180°, shown in Fig. 8b, the upper leaf 
56b no longer supports the carrier and instead the car- 
rier has dropped slightly and is supported by the lower 
leaf 56c. Fig. 8c shows the position ofthe cam 56 rotated 
270° wherein the upper leaf 56b has rotated sufficiently 
to begin to engage the undercut 66 ofthe next slide car- 
rier while the opposite facing lower leaf 56c still supports 
the bottom carrier. After a full rotation of 360° as shown 
in Fig. 8d, the lower leaf 56c has rotated opposite the 



carrier stack and no longer supports the bottom carrier 
which now rests on the loading platform 52. At the same 
position, the upper leaf 56b supports the next carrier for 
repeating the cycle, 
s [0032] Referring again to Figs. 7a and 7b, when the 
carrier drops to the loading platform 52, the contact dos- 
es switch 90 which activates motors 86 and 87. Motor 
86 drives the infeed drive belt 59 until the pusher tab 58 
makes contact with the carrier and pushes the carrier 
to onto the X-Y stage drive belt 50. The stage drive belt 50 
advances the carrier until contact is made with switch 
91, the closing of which begins the slide scanning proc- 
ess described further herein. Upon completion of the 
scanning process, theX-Ystage 38 moves to an unload 
is position and motors 87 and 88 are activated to transport 
the carrierto the unloading platform 36 using stage drive 
belt 50. As noted, motor 88 drives outfeed gear 93 to 
engage the carrier pitch rack 68 of the carrier 60 (Fig. 
6b) until switch 92 is contacted. Closing switch 92 acti- 
20 vates motor 89 to rotate the unloading platform 36. 
[0033] The unloading operation is shown in more de- 
tail in end views ofthe output module 17 (Figs. 9a-9d). 
In Fig. 9a, the unloading platform 36 is shown in a hor- 
izontal position supporting a slide carrier 60. The hold- 
25 down mechanism 94 secures the carrier 60 at one end. 
Fig. 9b shows the output module 17 after motor 89 has 
rotated the unloading platform 36 to a vertical position, 
at which point the spring loaded hold-down mechanism 
94 releases the slide carrier 60 into the output hopper 
so 18. The carrier 60 is supported in the output hopper 18 
by means of ears 64 (Figs. 6a and 6b). Fig. 9c shows 
the unloading platform 36 being rotated back towards 
the horizontal position. As the platform 36 rotates up- 
ward, it contacts the deposited carrier 60 and the up- 
35 ward movement pushes the carrier toward the front of 
the output hopper 18. Fig. 9d shows the unloading plat- 
form 36 at its original horizontal position after having out- 
put a series of slide carriers 60 to the output hopper 1 8. 
Having described the overall system and the automated 
40 slide handling feature, the aspects ofthe apparatus 10 
relating to scanning, focusing and image processing will 
now be described in further detail. 
[0034] In some cases, an operator will know ahead of 
time where the scan area of interest is on the slide. Con- 
■*5 ventional preparation of slides for examination provides 
repeatable and known placement ofthe sample on the 
slide. The operator can therefore instruct the system to 
always scan the same area at the same location of every 
slide which is prepared in this fashion. But there are oth- 
50 er times in which the area of interest is not known, for 
example, where slides are prepared manually with a 
known smear technique. One feature of the invention 
automatically determines the scan area using a texture 
analysis process. 
55 [0035] Fig. 10 is a flow diagram that describes the 
processing associated with the automatic location of a 
scan area. As shown in this figure, the basic method is 
to pre-scan the entire slide area to determine texture 
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features that indicate the presence of a smear and to 
discriminate these areas from dirt and other artifacts. 
[0036] At each location of this raster scan, an image 
such as in Fig. 12 is acquired and analyzed for texture 
information at steps 204 and 206. Since it is desired to 
locate the edges of the smear sample within a given im- 
age, texture analyses are conducted over areas called 
windows 78, which are smaller than the entire image as 
shown in Fig. 12. The process iterates the scan across 
the slide at steps 208, 21 0, 212 and 214. 
[0037] In the interest of speed, the texture analysis 
process is performed at a lower magnification, prefera- 
bly at a 4x objective. One reason to operate at low mag- 
nification is to image the largest slide area at any one 
time. Since cells do not yet need to be resolved at this 
stage of the overall image analysis, the 4x magnification 
is preferred. On a typical slide, as shown in Fig. 11, a 
portion 72b of the end of the slide 72 is reserved for la- 
beling with identification information. Excepting this la- 
bel area, the entire slide is scanned in a rasterscan fash- 
ion 75 to yield a number of adjacent images. 
[0038] Texture values for each window include the 
pixel variance over a window, the difference between 
the largest and smallest pixel value within a window, and 
other indicators. The presence of a smear raises the tex- 
ture values compared with a blank area. 
[0039] One problem with a smear from the standpoint 
of determining its location is its non-uniform thickness 
and texture. For example, the smear is likely to be rela- 
tively thin at the edges and thicker towards the middle 
due to the nature of the smearing process. To accom- 
modate forthe non-uniformity, texture analysis provides 
a texture value for each analyzed area. The texture val- 
ue fends to gradually rise as the scan proceeds across 
a smear from a thin area to a thick area, reaches a peak, 
and then falls off again to a lower value as a thin area 
at the edge is reached. The problem is then to decide 
from the series of texture values the beginning and end- 
ing, or the edges, of the smear. The texture values are 
fit to a square wave waveform since the texture data 
does not have sharp beginnings and endings. 
[0040] After conducting this scanning and texture 
evaluation operation, one must determine which areas 
of elevated texture values represent the desired smear 
74, and which represent undesired artifacts. This is ac- 
complished by fitting a step function, on a line by line 
basis to the texture values in step 216. This function, 
which resembles a single square wave across the 
smear with a beginning at one edge, and end at the other 
edge, and an amplitude provides the means for discrim- 
ination. The amplitude of the best-fit step function is uti- 
lized to determine whether smear or dirt is present since 
relatively high values indicate smear. If it is decided that 
smear is present, the beginning and ending coordinates 
of this pattern are noted until all lines have been proc- 
essed, and the smear sample area defined at 218. 
[0041] After an initial focusing operation described 
further herein, the scan area of interest is scanned to 



acquire images for image analysis. The preferred meth- 
od of operation is to initially perform a complete scan of 
the slide at low magnification to identify and locate can- 
didate objects of interest, followed by further image 
s analysis of the candidate objects of interest at high mag- 
nification in order to confirm the objects as celis. An al- 
ternate method of operation is to perform high magnifi- 
cation image analysis of each candidate object of inter- 
est immediately after the object has been identified at 
io low magnification. The low magnification scanning then 
resumes, searching for additional candidate objects of 
interest. Since it takes on the order of a few seconds to 
change objectives, this alternate method of operation 
would take longer to complete. 
is [0042] The operator can pre-select a magnification 
level to be used forthe scanning operation. A low mag- 
nification using a 10x objective is preferred forthe scan- 
ning operation since a larger area can be initially ana- 
lyzed for each acquired scan image. The overall detec- 
20 tion process for a cell includes a combination of deci- 
sions made at both low (10x) and high magnification 
(40x) levels. Decision making at the 10x magnification 
level is broader in scope, i.e., objects that loosely fit the 
relevant color, size and shape characteristics are iden- 
25 tified at the 1 0x level. Analysis at the 40x magnification 
level then proceeds to refine the decision making and 
confirm objects as likely cells or candidate objects of in- 
terest. For example, at the 40x level it is not uncommon 
to find that some objects that were identified at 1 0x are 
30 artifacts which the analysis process will then reject. In 
addition, closely packed objects of interest appearing at 
10x are separated at the 40x level. 
[0043] In a situation where a cell straddles or overlaps 
adjacent image fields, image analysis of the individual 
35 adjacent image fields could result in the cell being re- 
jected or undetected. To avoid missing such cells, the 
scanning operation compensates by overlapping adja- 
cent image fields in both the x and y directions. An over- 
lap amount greater than half the diameter of an average 
40 cell is preferred, in the preferred embodiment, the over- 
lap is specified as a percentage of the image field in the 
x and y directions. 

[0044] The time to complete an image analysis can 
vary depending upon the size of the scan area and the 
& number of candidate cells, or objects of interest identi- 
fied. For one example, a complete image analysis of a 
scan area of two square centimeters in which 50 objects 
of interest are confirmed can be performed in about 12 
to 1 5 minutes. This example includes not only focusing, 
50 scanning and image analysis but also the saving of 40x 
images as a mosaic on hard drive 21 (Fig. 2). 
[0045] Consider the utility of the present invention in 
a "rare event" application where there may be one, two 
or a very small number of ceils of interest located some- 
55 where on the slide. To illustrate the nature ofthe problem 
by analogy, if one were to scale a slide to the size of a 
football field, a tumor cell, for example, would be about 
the size of a bottle cap. The problem is then to rapidly 
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search the football field and find the very small number 
of bottle caps and have a high certainty that none have 
been missed. 

[0046] Howeverthe scan area is defined, an initial fo- 
cusing operation must be performed on each slide prior 
to scanning. This is required since slides differ, in gen- 
eral, in their placement in a carrier. These differences 
include slight (but significant) variations of tilt of the slide 
in its carrier. Since each slide must remain in focus dur- 
ing scanning, the degree of tilt of each slide must be 
determined. This is accomplished with an initial focusing 
operation that determines the exact degree of tilt, so that 
focus can be maintained automatically during scanning. 
[0047] The initial focusing operation and other focus- 
ing operations to be described later utilize a focusing 
method based on processing of images acquired by the 
system. This method was chosen for its simplicity over 
other methods including use of IR beams reflected from 
the slide surface and use of mechanical gauges. These 
other methods also would not function properly when 
the specimen is protected with a coverglass. The pre- 
ferred method results in lower system cost and im- 
proved reliability since no additional parts need be in- 
cluded to perform focusing. 

[0040] Fig, 13A provides a flow diagram describing 
the "focus point" procedure. The basic method relies on 
the fact that the pixel value variance (or standard devi- 
ation) taken about the pixel value mean is maximum at 
best focus. A "brute-force" method could simply step 
through focus, using the computer controlled Z, or focus 
stage, calculate the pixel variance at each step, and re- 
turn to the focus position providing the maximum vari- 
ance. Such a method would be too time consuming. 
Therefore, additional features were added as shown in 
Fig. 13A. 

[0049] These features include the determination of 
pixel variance at a relatively coarse number of focal po- 
sitions, and then the fitting of a curve to the data to pro- 
vide a faster means of determining optimal focus. This 
basic process is applied in two steps, coarse and fine. 
[0050] During the coarse step at 220-230, the Z stage 
is stepped over a user-specified range of focus posi- 
tions, with step sizes that are also user-specified. It has 
been found that for coarse focusing, these data are a 
close fit to a Gaussian function. Therefore, this initial set 
of variance versus focus position data are least-squares 
fit to a Gaussian function at 228. The location of the peak 
of this Gaussian curve determines the initial or coarse 
estimate of focus position for input to step 232. Follow- 
ing this, a second stepping operation 232-242 is per- 
formed utilizing smallersteps over a smaller focus range 
centered on the coarse focus position. Experience indi- 
cates that data taken overthis smaller range are gener- 
ally best fit by a second order polynomial. Onceihis least 
squares fit is performed at 240, the peak of the second 
order curve provides the fine focus position at 244. 
[0051] Fig. 14 illustrates a procedure for how this fo- 
cusing method is utilized to determine the orientation of 



a slide in its carrier. As shown, focus positions are de- 
termined, as described above, for a 3 x 3 grid of points 
centered on the scan area at 264. Should one or more 
of these points lie outside the scan area, the method 

s senses at 266 this by virtue of low values of pixel vari- 
ance. In this case, additional points are selected closer 
to the center of the scan area. Fig. 15 shows the initial 
array of points 80 and new point 82 selected closer to 
the center. Once this array of focus positions is deter- 

10 mined at 268, a least squares plane is fit to this data at 
270. Focus points lying too far above or below this best- 
fit plane are discarded at 272 (such as can occur from 
a dirty cover glass over the scan area), and the data is 
then refit. This plane at 274 then provides the desired Z 

■is position information for maintaining focus during scan- 
ning. 

[0052] After determination of the best-fit focus plane, 
the scan area is scanned in an X raster scan over the 
scan area as described earlier. During scanning, theX 

20 stage is positioned to the starting point of the scan area, 
the focus (Z) stage is positioned to the best fit focus 
plane, an image is acquired and processed as described 
later, and this process is repeated for all points overthe 
scan area. In this way, focus is maintained automatically 

25 without the need for time-consuming refocusing at 
points during scanning. 

[0053] Prior to confirmation of cell objects at a 40x or 
60x level, a refocusing operation is conducted since the 
use of this higher magnification requires more precise 

30 focus than the best-fit plane provides. Fig. 16 provides 
the flow diagram for this process. As may be seen, this 
process is similar to the fine focus method described 
earlier in that the object is to maximize the image pixel 
variance. This is accomplished by stepping through a 

35 range of focus positions with the Z stage at 276, 278, 
calculating the image variance at each position at 278, 
fitting a second order polynomial to these data at 282, 
and calculating the peak of this curve to yield an esti- 
mate of the best focus position at 284, 286. This final 

40 focusing step differs from previous ones in thatthe focus 
range and focus step sizes are smaller since this mag- 
nification requires focus settings to within 0.5 micron or 
better. 

[0054] It should be noted that for some combinations 
45 of cell staining characteristics, improved focus can be 
obtained by numerically selecting the focus position that 
provides the largest variance, as opposed to selecting 
the peak of the polynomial. In such cases, the polyno- 
mial is used to provide an estimate of best focus, and a 
50 final step selects the actual Z position giving highest pix- 
el variance. It should also be noted that if at any time 
during the focusing process at 40x or60x the parame- 
ters indicate that the focus position is inadequate, the 
system automatically reverts to a coarse focusing proc- 
55 ess as described above with reference to Fig. 1 3A. This 
ensures that variations in specimen thickness can be 
accommodated in an expeditious manner. 
[0055] For some biological specimens and stains, the 
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focusing methods discussed above do not provide op- 
timal focused results. For example, certain white blood 
cells known as neutrophils may be stained with Fast 
Red, a commonly known stain, to identify alkaline phos- 
phatase in the cytoplasm of the cells. To further identify 
these cells and the material within them, the specimen 
may be counterstained with hemotoxylin to identify the 
nucleus of the cells. In cells so treated, the cytoplasm 
bearing alkaline phosphatase becomes a shade of red 
proportionate to the amount of alkaline phosphatase in 
the cytoplasm and the nucleus becomes blue. However, 
where the cytoplasm and nucleus overlap, the cell ap- 
pears purple. These color combinations appear to pre- 
clude the finding of a focused Z position using the focus 
processes discussed above. 

[0056] In an effort to find a best focal position at high 
magnification, a focus method, such as the one shown 
in Fig. 13B, may be used. That method begins by se- 
lecting a pixel near the center of a candidate object of 
interest (Block 248) and defining a region of interest cen- 
tered about the selected pixel (Block 250). Preferably, 
the width of the region of interest is a number of columns 
which is a power of 2. This width preference arises from 
subsequent processing of the region of interest prefer- 
ably using a one dimensional Fast Fourier Transform 
(FFT) technique. As is well known within, the art, 
processing columns of pixel values using the FFT tech- 
nique is facilitated by making the number of columns to 
be processed a power of two. While the height of the 
region of interest is also a power of two in the preferred 
embodiment, it need not be unless a two dimensional 
FFT technique is used to process the region of interest. 
After the region of interest is selected, the columns of 
pixel values are processed using the preferred one di- 
mensional FFT to determine a spectra of frequency 
components for the region of interest (Block 252). The 
frequency spectra ranges from DC to some highest fre- 
quency component. For each frequency component, a 
complex magnitude is computed. Preferably, the com- 
plex magnitudes for the frequency components which 
range from approximately 25% of the highest compo- 
nent to approximately 75% of the highest component 
are squared and summed to determine the total power 
for the region of interest (Block 254). Alternatively, the 
region of interest may be processed with a smoothing 
window, such as a Hanning window, to reduce the spu- 
rious high frequency components generated by the FFT 
processing of the pixel values in the region of interest. 
Such preprocessing of the region of interest permits all 
complex magnitude over the complete frequency range 
to be squared and summed. Afterthe power for a region 
has been computed and stored (Block 256), a new focal 
position is selected, focus adjusted (Blocks 258, 260), 
and the process repeated. After each focal position has 
been evaluated, the one having the greatest power fac- 
tor is selected as the one best in focus (Block 262). 
[0057] The following describes the image processing 
methods which are utilized to decide whether a candi- 



date object of interest such as a stained tumor ceil is 
present in a given image, or field, during the scanning 
process. Candidate objects of interest which are detect- 
ed during scanning are reimaged at higher (40x or60x) 
5 magnification, the decision confirmed, and a region of 
interest for this ceil saved for later review by the pathol- 
ogist. 

[0058] The image processing includes color space 
conversion, low pass filtering, background suppression, 
w artifact suppression, morphological processing, and 
blob analysis. One or more of the latter steps can op- 
tionally be eliminated. The operator is provided with an 
option to configure the system to perform any or all of 
these steps and whetherto perform certain steps more 
is than once or several times in a row. It should also be 
noted that the sequence of steps may be varied and 
thereby optimized for specific reagents or reagent com- 
binations; however, the sequence described herein is 
preferred. It should be noted that the image processing 
20 steps of low pass filtering, thresholding, morphological 
processing, and blob analysis are generally known im- 
age processing building blocks. 
[0059] An overview of the preferred process is shown 
in Fig. 17A The preferred process for identifying and 
25 locating candidate objects of interest in a stained bio- 
logical specimen on a slide begins with an acquisition 
of images obtained by scanning the slide at low magni- 
fication (Block 288). Each image is then converted from 
a first color space to a second color space (Block 290) 
so and the color converted image is low pass filtered (Block 
292). The pixels of the low pass filtered image are then 
compared to a threshold (Block 294) and, preferably, 
those pixels having a value equal to or greater than the 
threshold are identified as candidate object of interest 
35 pixels and those less than the threshold are determined 
to be artifact or background pixels. The candidate object 
of interest pixels are then morphologically processed to 
identify groups of candidate object of interest pixels as 
candidate objects of interest (Block 296). These Candi- 
da date objects of interest are then compared to blob anal- 
ysis parameters (Block 298) to further differentiate can- 
didate objects of interest from objects which do not con- 
form to the blob analysis parameters and, thus, do not 
warrant further processing. The location of the candi- 
45 date objects of interest may be stored prior to confirma- 
tion at high magnification. The process continues by de- 
termining whetherthe candidate objects of interest have 
been confirmed (Block 300). If they have not been con- 
firmed, the optical system is set to high magnification 
so (Block 302) and images of the slide at the locations cor- 
responding to the candidate objects of interest identified 
in the low magnification images are acquired (Block 
288). These images are then color converted (Block 
290), low pass filtered (Block 292), compared to a 
55 threshold (Block 294), morphologically processed 
(Block 296), and compared to blob analysis parameters 
(Block 298) to confirm which candidate objects of inter- 
est located from the low magnification images are ob- 
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jects of interest. The coordinates of the objects of inter- 
est are then stored for future reference (Block 303). 
[0060] Neural net processing schemes were not con- 
sidered for the preferred embodiment for several rea- 
sons. Firstly, the preferred embodiment is optimized for 
"rare-event" detection, although it is not limited to this 
case. Since neural nets must be trained on what to look 
for, sometimes several thousands Of examples must be 
presented to the neural net for this training. This is im- 
practical for a rare-event application. Secondly, neural 
net processing can be sfowerthan "deterministic" meth- 
ods, sometimes by large factors. Therefore, neural nets 
were not deemed appropriate for this application, al- 
though certain features of the invention may be advan- 
tageously applied to neural network systems. 
[0061] In general, the candidate objects of interest, 
such as tumor cells, are detected based on a combina- 
tion of characteristics, including size, shape, and color. 
The chain of decision making based on these charac- 
teristics preferably begins with a color space conversion 
process. The CCD camera coupled to the microscope 
subsystem outputs a color image comprising a matrix of 
640 x 480 pixels. Each pixel comprises red, green and 
blue (RGB) signal values. 

[0062] !t is desirable to transform the matrix of RGB 
values to a different color space because the difference 
between candidate objects of interest and their back- 
ground, such as tumor and normal cells, may be deter- 
mined from their respective colors. Specimens are gen- 
erally stained with one or more industry standard stains 
(e.g., DAB, New Fuchsin, AEC) which are "reddish" in 
color. Candidate objects of interest retain more of the 
stain and thus appear red while normal cells remain un- 
stained. The specimens may also be counterstained 
with hematoxalin so the nuclei of normal cells or cells 
not containing an object of interest appear blue. In ad- 
dition to these objects, dirt and debris can appear as 
black, gray, or can also be lightly stained red or blue de- 
pending on the staining procedures utilized. The resid- 
ual plasma or other fluids also present on a smear may 
also possess some color. 

[0063] In the color conversion operation, a ratio oftwo 
of the RGB signal values is formed to provide a means 
for discriminating color information. With three signal 
valuesforeach pixel, nine different ratios can be formed: 

R/R, R/G, R/B, G/G, G/B, G/R, B/B, B/G, B/R 
[0064] The optimal ratio to select depends upon the 
range of color information expected in the slide speci- 
men. As noted above, typical stains used for detecting 
candidate objects of interest such as tumor cells are pre- 
dominantly red, as opposed to predominantly green or 
blue. Thus, the pixels of a cell of interest which has been 
stained contain a red component which is larger than 
either the green or blue components. A ratio of red di- 
vided by blue (R/B) provides a value which is greater 
than one fortumor cells but is approximately one for any 
clear or white areas on the slide. Since the remaining 
cells, i.e., normal cells, typically are stained blue, the Rl 



B ratio for pixels of these latter cells yields values of less 
than one. The R/B ratio is preferred for clearly separat- 
ing the color information typical in these applications. 
[0065] Fig. 17B illustrates the flow diagram by which 
5 this conversion is performed. In the interest of process- 
ing speed, the conversion is implemented with a look up 
table. The use of a look up table for color conversion 
accomplishes three functions: 1) performing a division 
operation; 2) scaling the result for processing as an im- 
10 age having pixel values ranging from 0 to 255; and 3) 
defining objects which have low pixel values in each 
color band (R,G,B) as "black" to avoid infinite ratios (i. 
e., dividing by zero). These "black" objects are typically 
staining artifacts or can be edges of bubbles caused by 
15 pasting a coverglass over the specimen. 

[0066] Once the look up table is built at 304 for the 
specific color ratio (i.e., choices oftumorand nucleated 
cell stains), each pixel in the original RGB image is con- 
verted at 308 to produce the output. Since it is of interest 
20 to separate the red stained tumor cells from blue stained 
normal ones, the ratio of color values is then scaled by 
a user specified factor. As an example, for a factor of 
128 and the ratio of (red pixel value)/(blue pixel value), 
clear areas on the slide would have a ratio of 1 scaled 
25 by 128 for a final X value of 128. Pixels which lie in red 
stained tumor cells would have X value greater than 
128, while blue stained nuclei of norma! cells would have 
value less than 128. In this way, the desired objects of 
interest can be numerically discriminated. The resulting 
30 640 x 480 pixel matrix, referred to as theX-image, is a 
gray scale image having values ranging from 0 to 255. 
[0067] Other methods exist for discriminating color in- 
formation. One classical method converts the RGB color 
information into another color space, such as HSI (hue, 
35 saturation, intensity) space. In such a space, distinctly 
different hues such as red, blue, green, yellow, may be 
readily separated. In addition, relatively lightly stained 
objects may be distinguished from more intensely 
stained ones by virtue of differing saturations. However, 
40 converting from RGB space to HSI space requires more 
complex computation. Conversion to a color ratio is fast- 
er; for example, a full image can be converted by the 
ratio technique of the present invention in about 30 ms 
while an HSI conversion can take several seconds. 
45 [0068] In yet another approach, one could obtain color 
information by taking a single color channel from the 
camera. As an example, consider a blue channel, in 
which objects that are red are relatively dark. Objects 
which are blue, or white, are relatively light in the blue 
50 channel. In principle, one could take a single color chan- 
nel, and simply set a threshold wherein everything dark- 
er than some threshold is categorized as a candidate 
object of interest, for example, a tumor cell, because it 
is red and hence dark in the channel being reviewed. 
55 However, one problem with the single channel approach 
occurs where illumination is not uniform. Non-uniformity 
of illumination results in non-uniformity across the pixel 
values in any color channel, for example, tending to 
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peak in the middle of the image and dropping off at the 
edges where the illumination falls off. Performing 
thresholding on this non-uniform color information runs 
into problems, as the edges sometimes fall below the 
threshold, and therefore it becomes more difficultto pick 
the appropriate threshold level. However, with the ratio 
technique, if the values of the red channel fall off from 
center to edge, then the values of the blue channel also 
fall off center to edge, resulting in a uniform ratio. Thus, 
the ratio technique is more immune to illumination non- 
uniformities. 

[0069] As previously described, the color conversion 
scheme is relatively insensitive to changes in color bal- 
ance, i.e., the relative outputs ofthe red, green, and blue 
channels. However, some control is necessary to avoid 
camera saturation, or inadequate exposures in any one 
ofthe color bands. This color balancing is performed au- 
tomatically by utilizing a calibration slide consisting of a 
clear area, and a "dark" area having a known optical 
transmission or density. The system obtains images 
from the clear and "dark" areas, calculates "white" and 
"black" adjustments for the image processor 25, and 
thereby provides correct color balance. 
[0070] In addition to the color balance control, certain 
mechanical alignments are automated in this process. 
The center point in the field of view for the various mi- 
croscope objectives as measured on the slide can vary 
by several (or several tens of) microns. This is the result 
of slight variations in position of the microscope objec- 
tives 44a as determined by the turret 44 (Fig. 4), smali 
variations in alignment ofthe objectives with respect to 
the system optical axis, and other factors. Since it is de- 
sired that each microscope objective be centered at the 
same point, these mechanical offsets must be meas- 
ured and automatically compensated. 
[0071] This is accomplished by imaging a test slide 
which contains a recognizable feature or mark. An im- 
age of this pattern is obtained bythesystemwithagiven 
objective, and the position ofthe mark determined. The 
system then rotates the turret to the next lens objective, 
obtains an image ofthe test object, and its position is 
redetermined. Apparent changes in position of the test 
markare recorded forthis objective. This process is con- 
tinued for all objectives.. 

[0072] Once these spatial offsets have been deter- 
mined, they are automatically compensated for by mov- 
ing the stage 38 by an equal (but opposite) amount of 
offset during changes in objective. In this way, as differ- 
ent lens objectives are selected, there is no apparent 
shift in center point or area viewed. 
[0073] A low pass filtering process precedes thresh- 
olding. An objective of thresholding is to obtain a pixel 
image matrix having only candidate objects of interest, 
such as tumor cells above a threshold level and every- 
thing else below it. However, an actual acquired image 
will contain noise. The noise can take several forms, in- 
cluding white noise and artifacts. The microscope slide 
can have small fragments of debris that pick up color in 



the staining process and these are known as artifacts. 
These artifacts are generally small and scattered areas, 
on the order of a few pixels, which are above the thresh- 
old. The purpose of low pass filtering is to essentially 
s blur or smear the entire color converted image. The low 
pass filtering process will smear artifacts more than larg- 
er objects of interest, such as tumor cells and thereby 
eliminate or reduce the number of artifacts that pass the 
thresholding process. The result is a cleaner threshold- 

io ed image downstream. 

[0074] In the low pass filter process, a 3 x 3 matrix of 
coefficients is applied to each pixel in the 640 x 480 x- 
image. A preferred coefficient matrix is as follows; 
[0075] At each pixel location, a 3x 3 matrix comprising 

is the pixel of interest and its neighbors is multiplied by the 
coefficient matrix and summed to yield a single value for 
the pixel of interest. The output of this spatial convolu- 
tion process is again a 640 x 480 matrix. 
[0076] As an example, consider a case where the 

20 center pixel and only the center pixel, has a value of 255 
and each of its other neighbors, top left, top, top right 
and so forth, have values of 0. This singular white pixel 
case corresponds to a small object. The result of the 
matrix multiplication and addition using the coefficient 

2S matrix is a value of 1/9 (255) or 28 for the center pixel, 
a value which is below the nominal threshold of 128. 
Now consider another case in which all the pixels have 
a value of 255 corresponding to a large object. Perform- 
ing the low pass filtering operation on a 3 x 3 matrix for 

30 this case yields a value of 255 for the center pixel. Thus, 
large objects retain their values while small objects are 
reduced in amplitude or eliminated. In the preferred 
method of operation, the low pass filtering process is 
performed on the X image twice in succession. 

35 [0077] In order to separate objects of interest, such 
as a tumor cell in the x image from other objects and 
background, a thresholding operation is performed de- 
signed to set pixels within cells of interest to a value of 
255, and all other areas to 0. Thresholding ideally yields 

40 an image in which cells of interest are white and the re- 
mainder ofthe image is black. A problem one faces in 
thresholding is where to set the threshold level. One 
cannot simply assume that cells of interest are indicated 
by any pixel value above the nominal threshold of 128. 

45 A typical imaging system may use an incandescent hal- 
ogen light bulb as a light source. As the bulb ages, the 
relative amounts of red and blue output can change. The 
tendency as the bulb ages is forthe blue to drop off more 
than the red and the green. To accommodate for this 

so light source variation overtime, a dynamic thresholding 
process is used whereby the threshold is adjusted dy- 
namically for each acquired image. Thus, for each 640 
x 480 image, a single threshold value is derived specific 
to that image. 

ss [0078] As shown in Fig. 18, the basic method is to cal- 
culate, for each field, the mean lvalue, and the stand- 
ard deviation about this mean at 312. The threshold is 
then set at 314 to the mean plus an amount defined by 
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the product of a (user specified) factor and the standard 
deviation of the color converted pixel values. The stand- 
ard deviation correlates to the structure and number of 
objects in the image. Preferably, the user specified fac- 
tor is in the range of approximately 1.5 to 2.5. The factor 
is selected to be in the lower end of the range for slides 
in which the stain has primarily remained within cell 
boundaries and the factor is selected to be in the upper 
end of the range for slides in which the stain is perva- 
sively present throughout the slide. In this way, as areas 
are encountered on the slide with greater or lower back- 
ground intensities, the threshold may be raised or low- 
ered to help reduce background objects. With this meth- 
od, the threshold changes in step with the aging of the 
light source such that the effects of the aging are can- 
celled out. The image matrix resulting at 316 from the 
thresholding step is a binary image of bJack (O) and 
white (255) pixels. 

[0079] As is often the case with thresholding opera- 
tions such as that described above, some undesired ar- 
eas will lie above the threshold value due to noise, small 
stained cell fragments, and other artifacts. It is desired 
and possible to eliminate these artifacts by virtue of their 
small size compared with legitimate cells of interest. 
Morphological processes are utilized to perform this 
function. 

[0080] Morphological processing is similar to the low 
pass filter convolution process described earlier except 
that it is applied to a binary image. Similarto spatial con- 
volution, the morphological process traverses an input 
image matrix, pixel by pixel, and places the processed 
pixels in an output matrix. Rather than calculating a 
weighted sum of neighboring pixels as in the low pass 
convolution process, the morphological process uses 
set theory operations to combine neighboring pixels in 
a nonlinear fashion. 

[0O81] Erosion is a process whereby a single pixel lay- 
er is taken away from the edge of an object. Dilation is 
the opposite process which adds a single pixel layer to 
the edges of an object. The power of morphological 
processing is that it provides for further discrimination 
to eliminate small objects that have survived the thresh- 
olding process and yet are not likely tumor cells. The 
erosion and dilation processes that make up a morpho- 
logical "open" preferably make small objects disappear 
yet allows large objects to remain. Morphological 
processing of binary images is described in detail in 
"Digital Image Processing", pages 127-137, G.A. Bax- 
es, John Wiley & Sons, (1994). Fig. 19 illustrates the 
flow diagram for this process. As shown here, a mor- 
phological "open" process performs this suppression. A 
single morphological open consists of a single morpho- 
logical erosion 320 followed by a single morphological 
dilation 322. Multiple "opens" consist of multiple ero- 
sions followed by multiple dilations. In the preferred em- 
bodiment, one or two morphological opens are found to 
be suitable. 

[0082] At this point in the processing chain, the proc- 



essed image contains threshoided objects of interest, 
such as tumor cells (if any were present in the original 
image), and possibly some residual artifacts that were 
too large to be eliminated by the processes above. 

5 [0083] Fig. 20 provides a flow diagram illustrating a 
blob analysis performed to determine the number, size, 
and location of objects in the threshoided image. A blob 
is defined as a region of connected pixels having the 
same "color", in this case, a value of 255. Processing is 

10 performed over the entire image to determine the 
number of such regions at 324 and to determine the area 
and x,y coordinates for each detected blob at 326. 
[0084] Comparison of the size of each blob to a known 
minimum area at 328 fora tumor cell allows a refinement 

15 jn decisions about which objects are objects of interest, 
such as tumor cells, and which are artifacts. The location 
(x,y coordinates) of objects identified as cells of interest 
in this stage are saved for the final 40x reimaging step 
described below. Objects not passing the size test are 

20 disregarded as artifacts. 

[0085] The processing chain described above identi- 
fies objects at the scanning magnification as cells of in- 
terest candidates. As illustrated in Fig. 21, at the com- 
pletion of scanning, the system switches to the 40x mag- 

25 nification objective at 330, and each candidate is reim- 
aged to confirm the identification 332. Each 40x image 
is reprocessed at 334 using the same steps as de- 
scribed above but with test parameters suitably modified 
for the higher magnification (e.g. area). At 336, a region 

30 of interest centered on each confirmed cell is saved to 
the hard drive for review by the pathologist. 
[0086] As noted earlier, a mosaic of saved images is 
made available forviewing by the pathologist, As shown 
in Fig. 22, a series of images of cells which have been 

35 confirmed by the image analysis is presented in the mo- 
saic 150. The pathologist can then visually inspect the 
images to make a determination whetherto accept (1 52) 
or reject (153) each cell image. Such a determination 
can be noted and saved with the mosaic of images for 
generating a printed report. 

[0087] In addition to saving the image of the cell and 
its region, the cell coordinates are saved should the pa- 
thologist wish to directly view the cell throug h the oculars 
or on the image monitor. In this case, the pathologist 
45 reloads the slide carrier, selects the slide and cell for 
review from a mosaic of cell images, and the system 
automatically positionsthe cell underthe microscope for 
viewing. 

[0088] It has been found that normal cells whose nu- 
so clei have been stained with hematoxylin are often quite 
numerous, numbering in the thousands per 10x image. 
Since these cells are so numerous, and since they tend 
to clump, counting each individual nucleated cell would 
add an excessive processing burden, at the expense of 
55 speed, and would not necessarily provide an accurate 
count due to clumping. The apparatus performs an es- 
timation process in which the total area of each field that 
is stained hematoxylin blue is measured and this area 
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is divided by the average size of a nucleated cell. Fig. 
23 outlines this process. 

[0089] In this process, a single color band (the red 
channel provides the best contrast for blue stained nu- 
cleated cells) is processed by calculating the average 
pixel value for each field at 342, establishing two thresh- 
old values (high and low) as indicated at 344, 346, and 
counting the number of pixels between these two values 
at 348. In the absence of dirt, or other opaque debris, 
this provides a count of the number of predominantly 
blue pixels. By dividing this value by the average area 
for a nucleated cell at 350, and looping over all fields at 
352, an approximate cell count is obtained. Preliminary 
testing of this process indicates an accuracy with +/- 
15%. It should be noted that for some slide preparation 
techniques, the size of nucleated cells can be signifi- 
cantly larger than the typical size. The operator can se- 
lect the appropriate nucleated cell size to compensate 
for these characteristics. 

[0090] As with any imaging system, there is some loss 
of modulation transfer (i.e. contrast) due to the modula- 
tion transfer function (MTF) characteristics of the imag- 
ing optics, camera, electronics, and other components. 
Since it is desired to save "high quality" images of cells 
of interest both for pathologist review and for archival 
purposes, it is desired to compensate for these MTF 
losses. 

[0091] An MTF compensation, or MTFC, is performed 
as a digital process applied to the acquired digital imag- 
es. A digital filter is utilized to restore the high spatial 
frequency content of the images upon storage, while 
maintaining low noise levels. With this MTFC technolo- 
gy, image quality is enhanced, or restored, through the 
use of digital processing methods as opposed to con- 
ventional oil-immersion or other hardware based meth- 
ods. MTFC is described further in "The Image Process- 
ing Handbook," pages 225 and 337, J. C. Rues, CRC 
Press (1995). 

[0092] Referring to Fig. 24, the functions available in 
a user interface of the apparatus 10 are shown. From 
the user interface, which is presented graphically on 
computer monitor26, an operator can select among ap- 
paratus functions which include acquisition 402, ana- 
lysts 404, and system configuration 406. At the acquisi- 
tion level 402, the operator can select between manual 
408 and automatic 410 modes of operation. In the man- 
ual mode, the operator is presented with manual oper- 
ations 409. Patient information 414 regarding an assay 
can be entered at 412. 

[0093] In the analysis level 404, review 416 and report 
418 functions are made available. At the review level 
416, the operator can select a montage function 420. At 
this montage level, a pathologist can perform diagnostic 
review functions including visiting an image 422, accept/ 
reject of cells 424, nucleated cell counting 426, accept/ 
reject of cell counts 428, and saving of pages at 430. 
The report level 41 8 allows an operator to generate pa- 
tient reports 432. 



[0094] In the configuration level 406, the operator can 
select to configure preferences at 434, input operator 
information 437 at 436, create a system log at 438, and 
toggle a menu panel at 440. The configuration prefer- 
s ences include scan area selection functions at 442, 452; 
montage specifications at 444, bar code handling at 
446, default cell counting at 448, stain selection at 450, 
and scan objective selection at 454. 

10 Equivalents 

[0095] While this invention has been particularly 
shown and described with references to preferred em- 
bodiments thereof, it will be understood by those skilled 
1$ in the art that various changes in form and details may 
be made therein without departing from the spirit and 
scope of the invention as defined by the appended 
claims. 



Claims 

1. A method for initially focusing an optical system of 
an automated microscopy system comprising the 

25 steps of 

(a) positioning an optical system at an initial Z 
stage position; 

(b) acquiring an image and calculating a pixel 
30 variance about a pixel mean for the acquired 

image; 

(c) incrementing the position of the Z stage; 

(d) repeating steps (b) and (c) for a fixed 
number of iterations; 

35 (e) performing a least squares fit of the variance 

data to a known function; and 
(f) selecting the peak value of the least squares 
fit curve as an estimate of the best focal posi- 
tion. 

40 

2. A method for initially focusing an optical system of 
an automated microscopy system comprising the 
steps of: 

« (a) positioning an optical system at an initial Z 

stage position; 

(b) acquiring at low magnification an image of 
a slide having a stained biological specimen 
thereon and calculating a pixel variance about 

50 a pixel mean for the acquired image; 

(c) incrementing the position of the Z stage; 

(d) repeating steps (b) and (c) for a fixed 
number of coarse iterations to form a first set 
of variance data; 

55 (e) performing a least squares fit of the first set 

of variance data to a first function; 
(f) positioning the Z stage at a position near the 
peak of the first function; 
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(g) repeating steps (b) and (c) for a fixed 
number of fine iterations to form a second set 
of variance data; 

(h) performing a least squares fit of the second 
set of variance data to a second function; 5 

(i) selecting the peak value of the least squares 
fit curve as an estimate of the best focal posi- 
tion; and 

(j) performing the above steps for an array of 
X-Y stage positions to form an array of focal po- to 
sitions and performing a least squares fit of the 
array of focal positions to yield a least squares 
fit focal plane. 

A method for focusing an optical system of an au- ts 
tomated microscopy system at high magnification 
comprising the steps of 

(a) positioning an optical system at an initial Z 
stage position; 20 

(b) acquiring an image and selecting a center 
pixel of a candidate object of interest; 

(c) defining a region of interest centered about 
the selected center pixel; 

(d) performing a fast fourier transform of said 25 
region of interest to identify frequency compo- 
nents for the region of interest and complex 
magnitudes for the frequency components; 

(e) computing a power value by summing the 
square of the complex magnitudes for the fre- 30 
quency components that are within the range 

of frequencies of 25% to 75% of a maximum 
frequency component forthe fast fourier trans- 
form of the region of interest; 

(f) incrementing the position of the Z stage; 35 

(g) repeating steps (b) - (e) for a fixed number 
of iterations; and 

(h) selecting theZ stage position corresponding 
to the largest power value as the best focal po- 
sition. 40 

A method for focusing an optical system of an au- 
tomated microscopy system at high magnification 
comprising the steps of: 

45 

(a) positioning an optical system at an initial Z 
stage position; 

(b) acquiring an image and selecting a center 
pixel of a candidate object of interest; 

(c) defining a region of interest centered about so 
the selected center pixel; 

(d) applying a Hanning window function to the 
region of interest; 

(d) performing a fast fourier transform of said 
region ofinterestfoilowing the application ofthe ss 
Hanning window function to identify frequency 
components forthe region of interest and com- 
plex magnitudes for the frequency compo- 



nents; 

(e) computing a power value by summing the 
square ofthe complex magnitudes for the fre- 
quency components for the fast fourier trans- 
form ofthe region of interest; 

(f) incrementing the position ofthe Z stage; 

(g) repeating steps (b) - (e) for a fixed number 
of iterations; and 

(h) selecting the Zstage position corresponding 
to the largest power value as the best focal po- 
sition. 

5. Apparatus for automatic image analysis of a slide 
having a biological specimen, comprising: 

an optical system having an X-Y stage; 
means for scanning over a scan area of the 
slide at a plurality of locations at low magnifica- 
tion ofthe optical system; 
means for acquiring a low magnification image 
at each location in the scan area; 
a processor for processing each low magnifica- 
tion image to detect candidate objects of inter- 
est; 

means for storing X-Y coordinates of each lo- 
cation for each candidate object of interest; 
means for adjusting the optical system to a high 
magnification; 

means for repositioning theX-Y stage to the lo- 
cation for each candidate object of interest; 
means for acquiring a high magnification image 
of each candidate object of interest; and 
a storage device for storing each high magnifi- 
cation image. 

6. Apparatus for automatic image analysis of a slide 
having a biological specimen, comprising: 

an optical system having an X-Y stage; 
means for scanning over a scan area of the 
siide at a plurality of locations at low magnifica- 
tion ofthe optical system; 
means for acquiring a low magnification image 
at each location in the scan area; 
a processor for processing each low magnifica- 
tion image to detect candidate objects of inter- 
est; 

means for storing X-Y coordinates of each lo- 
cation for each candidate object of interest; 
means for adjusting the optical systemto a high 
magnification; 

means for repositioning the X-Y stage to the 
center location for each candidate object of in- 
terest; 

means for acquiring a high magnification image 
of each candidate object of interest; and 
a storage device for storing each high magnifi- 
cation image. 
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POSITION FOCUS - (Z) STAGE TO USER-DEFINED Z 
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IF, AT ANY POINT AN ADEQUATE FOCUS DETERMINATION 

IS NOT MADE (DUE TO LACK OF IMAGE CONTENT), RE- 
POSITION STAGEHALF-WAY BETWEEN POINT AND CENTER 
OF SCAN AREA. REPEAT "FOCUS POINT" 
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ACQUIRE IMAGE 
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(USER SPECIFIED) COLOR PLANE 
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CALCULATE THRESH LO VALUE = (USER SPECIFIED 
LO FACTOR) * PIXEL AVERAGE 
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CALCULATE THRESH HI VALUE = (USER SPECIFIED 
HI FACTOR) * PIXEL AVERAGE 
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